Changes to lexer/parser on the hot path require benchmarking.

If you don't have a large codebase to do it on, we can measure it for you, but it will take time.

Some benchmarking after running on a ~1400 KLOC repository. This is 10 runs of .luau-compile --null . and reading the time for parsing that is printed (note: which is 2dp)

And decided to run in hyperfine if it's a helpful comparison:

$ hyperfine -N --warmup 1 "./luau-compile.old --null ." "./luau-compile.new --null --fflags=LuauLexerTokenizesWhitespace=False ." "./luau-compile.new --null --fflags=LuauLexerTokenizesWhitespace=True ."
Benchmark 1: ./luau-compile.old --null .
  Time (mean ± σ):      2.043 s ±  0.048 s    [User: 1.083 s, System: 0.648 s]
  Range (min … max):    1.947 s …  2.121 s    10 runs

Benchmark 2: ./luau-compile.new --null --fflags=LuauLexerTokenizesWhitespace=False .
  Time (mean ± σ):      2.062 s ±  0.066 s    [User: 1.091 s, System: 0.648 s]
  Range (min … max):    1.979 s …  2.172 s    10 runs

Benchmark 3: ./luau-compile.new --null --fflags=LuauLexerTokenizesWhitespace=True .
  Time (mean ± σ):      2.077 s ±  0.064 s    [User: 1.114 s, System: 0.648 s]
  Range (min … max):    2.026 s …  2.249 s    10 runs

Summary
  ./luau-compile.old --null . ran
    1.01 ± 0.04 times faster than ./luau-compile.new --null --fflags=LuauLexerTokenizesWhitespace=False .
    1.02 ± 0.04 times faster than ./luau-compile.new --null --fflags=LuauLexerTokenizesWhitespace=True .

If I keep running it though the numbers do fluctuate. Let me know if I should do something different to benchmark

Thanks, this makes sense, will take a look at this. For context, my original thinking was to make the lexer retain all source information so that given a stream of just lexer tokens, we could recreate the original input. Then, given an AST node and a set of related lexer tokens (would have to be stored during parsing), we'd be able to recreate exactly how that node was written.

Indeed, this would be important so that we could preserve the exact style of whitespace (spaces vs. tabs, leading/trailing whitespace, etc.).

But, the general idea of "filling in the gaps" with just location info is interesting. I did not explore this much, I'll try playing around with that further.

Or maybe we don't need to even build that structure and when we output the whitespace we can go and look what kind of symbols were used in the original file (even if AST nodes were moved around it should be reconstructable).

This would probably be good if the end goal was just to reconstruct the input, but I imagine if we did not build some sort of structure then the interface to perform code transforms would be a lot harder to work with, especially if you want to modify how the whitespace is. I can see it being doable though.